A generalized parsing framework for Abstract Grammars

This technical report presents a general framework for parsing a variety of grammar formalisms. We develop a grammar formalism, called an Abstract Grammar, which is general enough to represent grammars at many levels of the hierarchy, including Context Free Grammars (CFGs), Minimalist Grammars (MG), and other weakly MG-equivalent languages like Linear Context-Free Rewriting Systems. We then develop a single parsing framework which is capable of parsing grammars which are at least up to MGs on the hierarchy. Our parsing framework exposes a grammar interface modelled on the Abstract Grammar formalism, so that it can parse any particular grammar formalism that can be reduced to an Abstract Grammar. There is a great deal of previous work that is capable of parsing the grammars we treat here. All of the grammars mentioned here have parsers written specifically for them and there are frameworks more general than the one given here, such as probabilistic programming languages that can specify arbitrary probabilistic programs. Parsers for specific grammars are able to exploit optimizations specific to the task and formalism they were designed for and are therefore often faster than general systems. However, these parsers have the disadvantage that they cannot be used for formalisms other than the one that they were intended for, making it difficult to prototype and compare different formalisms. Our framework is the middle ground between these two approaches. We aim to be general enough to parse a variety of interesting formal grammars, while also exploiting optimizations specific to the parsing task. In the following, we define Abstract Grammars as a generalization of Context Free Grammars where (i) the rewrite rules are partial functions and (ii) the set of nonterminals is paired with a set of operations to form a heterogeneous algebra. We first define Abstract Context-free Grammars in §2 which incorporate property (i), and then define the fully general Abstract Grammars in §3 which incorporate property (ii). By (i), generalizing the rewrite rules to any partial function, we can group related CFG rewrite rules into a common function, allowing those rules to share probability mass. This is useful for representing a musical syntax where the the rules of prolongation and preparation, for example, are independent from the key (Rohrmeier & Neuwirth, 2015; Rohrmeier, 2011; Lerdahl & Jackendoff, 1985). By (ii) generalizing the nonterminals to elements of a heterogeneous algebra, we can represent languages higher than context-free on the hierarchy. This is important for representing natural language, which occupies the space of mildly context-sensitive languages (Shieber, 1985; Joshi, 1985). In §2, after defining Abstract Context-free Grammars, we present a reduction automaton upon abstract CFGs, which is used to state an abstract grammar interface. We then describe the parsing algorithm, including code fragments of our Julia1 implemetation. §3 defines Abstract Grammars and revises the reduction automaton and the grammar interface given in §2 for the fully general case. In §4, we show how to state specific interface functions for a Minimalist Grammar, and an implementation of this interface in Julia. ∗Corresponding author: Daniel Harasim [email protected]. 1julialang.org, see also Bezanson et al. (2017)